Microarrays and molecular research: noise discovery?

See Articles page 488 The promise of microarrays has been of apocalyptic dimensions. As put forth by one of their inventors, " all human illness can be studied by microarray analysis, and the ultimate goal of this work is to develop effective treatments or cures for every human disease by 2050 ". 1 All diseases are to be redefined, all human suffering reduced to gene-expression profiles. Cancer has been the most common early target of this revolution 2 and publications in the most prestigious journals have heralded the discovery of molecular signatures conferring different outcomes and requiring different treatments. Yet, in today's Lancet, Stefan Michiels and colleagues show that, on close scrutiny, in five of the seven largest studies on cancer prognosis, this technology performs no better than flipping a coin. The other two studies barely beat horoscopes. Why such failure? Give me information on a single gene and 200 patients, half of them dead, please. I bet I can show that this gene affects survival (pϽ0·05), even if it does not. One can do analyses: counting or ignoring exact follow-up; censoring at different timepoints; excluding specific causes of death; exploiting subgroup analyses; using dozens of different cutoffs to decide what constitutes inappropriate gene expression; and so forth. 3 Without highly specified a-priori hypotheses, there are hundreds of ways to analyse the dullest dataset. Thus, no matter what my discovery eventually is, it should not be taken seriously, unless it can be shown that the same exact mode of analysis gets similar results in a different dataset. Validation becomes even more important when datasets become complex and analytical options increase exponentially. Typically, patients are split into separate training and validation sets. In another common approach, each patient is left out in turn, a model is built, and then checked against the excluded patient. 4 Validation is still an analysis and can be manipulated as can any analysis. Several variants of inadequate or incomplete validation have been described. 2,5 Furthermore, when the same team does both the original analysis and validation thereof, one might consciously or unconsciously select the best-performing pair of training-validation data and analytical mode. Against this licence-to-analyse, one can use always and strictly the same method, generate randomly many training and validation sets, and examine whether results are stable. But then, as Michiels and colleagues show, the selected " important " genes rarely coincide across random replicates. Published …